A mixed reality technique for superposing a computer graphic image on a real scenery image as a background such that a user feels like a virtual object exists on the spot is known (for example, see Japanese Patent Application Laid-Open No. 11-088913 (counterpart US patent application is U.S. Pat. No. 6,522,312 (Oshima)). To realize realistic-sensations experience using this technique, it is important to enable the user's actual interaction such as touching or operating (feeling of touching or operating) the CG-drawn virtual object (hereinbelow, also simply referred to as a “virtual object”) in addition to superpose-display of CG image on the background real landscape image. To realize such interaction, it is necessary to display the user's hand(s) or the like (hereinbelow, referred to as a “subject(s)”) operating the virtual object in front of (foreground of) the virtual object. In a case where the user's hand(s) or the like which should be in the foreground of the CG virtual object is hidden with the virtual object, the sense of distance from the virtual object and the sense of reality break down, and the realism is impaired.
To solve this problem, the applicant has proposed in Japanese Patent Application Laid-Open No. 2003-296759a technique of preventing a foremost subject image from being hidden with a CG image. According to the technique, a background and a subject are obtained as real images, and color information on the subject to be displayed in front of the CG image (subject detection information) is manually registered in the system. Then CG drawing is prohibited in a pixel area having the registered color information. By using this technique, the subject as a foreground image can be displayed in front of the virtual object, without being hidden with the CG image, and the user can experience high-realism mixed reality.
However, in the above technique, as the registration of subject color information is manually performed, the registration cannot be easily performed by anyone but a worker with knowledge and experience. Further, as the subject area is determined only in accordance with color information, noise occurs due to erroneous recognition of subject area. That is, there is room for improvement.
From this technical background, there is a need for a technique of enabling registration of subject detection information with a comparatively simple operation which can be performed by a person without knowledge and experience such as a user or a system operator in place of manual registration of subject detection information by a worker with experience and knowledge.
Further, even if the subject color information can be registered by simple calibration, when a subject area is determined only in accordance with color information, an area in the background including a color similar to the subject's color is erroneously recognized as the subject. In this case, CG drawing in the area is prohibited and the background image appears. Further, in a case where the subject color information has changed due to change of illumination condition or the like, even an area inside the subject is erroneously recognized as a non-subject area, and a CG image is drawing on the subject. That is, in an area erroneously recognized as a subject area, CG drawing is prohibited and a background image which must be hidden is exposed. In an area erroneously recognized as a background (non-subject) area, a CG is drawn on the subject area in which CG drawing must be prohibited. Thus, reduction of noise due to erroneous recognition such as a background image in a CG image and a CG image in a subject is desired. Further, for accurate subject area detection, appropriate correction of subject detection information is also desired.